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Abstract 


This paper examines the current state of the literature on Integrated Planning and 
Advising for Student Success (iPASS), an increasingly popular approach to technology- 
mediated advising reform. We limit our analysis to sixty relevant documents that have 
been released since 2010. We categorize these items into four different groups based on 
their aim and rigor: (1) descriptive documents (39 items) that describe processes and 
challenges of iPASS implementation, (2) output reports (12 items) that examine usage 
data to better understand iPASS implementation, (3) correlational studies (5 items) that 
examine non-causal associations between different functionalities of iPASS tools and 
student outcomes, and (4) rigorous outcomes studies (4 items) that employ experimental 
or quasi-experimental methods to provide causal or near-causal estimations of iPASS- 
oriented interventions. iPASS is a relatively recent refonn approach, and while few 
studies are available that rigorously evaluate its effects, early findings suggest that 
individual components of iPASS interventions may have a positive impact on student 
outcomes, including persistence and credit accrual. This paper also offers suggestions on 
how colleges can use non-causal research to support and improve iPASS. 
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1. Introduction 


Colleges and universities are increasingly turning to technology to help provide 
more holistic support to students and to keep them on track to graduation. Over the past 
several years, an approach to technology-mediated student advising called Integrated 
Planning and Advising for Student Success (iPASS) has taken root. iPASS makes use of 
technology to promote, support, and sustain long-term, intrusive advising relationships. 
iPASS enables college personnel to engage in advising and student support relationships 1 
that (1) approach student support as a teaching function, (2) touch students on a regular 
basis, and (3) connect students to the information and services they need when they need 
them in order to keep students on track to program completion (Karp, Kalamkarian, 
Klempin, & Fletcher, 2016). 

Since 2012, the Bill & Melinda Gates Foundation has funded 42 colleges to 
implement the approach, and at least two state systems (Tennessee and North Carolina) 
have launched iPASS technologies at scale. iPASS is also an integral part of broader 
institutional redesign and reform efforts, including the American Association of 
Community Colleges (AACC) Pathways Project and the University Innovation Alliance. 
Additionally, more than 120 companies have launched iPASS-related products (Tyton 
Partners, 2014). As iPASS spreads and investment in advising technologies and student 
supports grow, it is important to assess the impact of this approach on student outcomes. 
To what extent does iPASS improve persistence and completion rates? Is iPASS an 
effective strategy to increase the numbers of students obtaining postsecondary 
credentials? 

One challenge in assessing the impact of iPASS is identifying or defining what 
constitutes iPASS. iPASS may look different at different institutions based on the 
college’s technology tools, their vision for student support, and the specific needs of their 
student population. Since colleges have different goals for iPASS, they might utilize 
iPASS tools differently. iPASS technology tools are most commonly used for three 
functions: (1) course or degree planning, to help students make suitable and accurate 

The notion of advising and student support that is intended here goes beyond academic advising and 
career planning and includes access to academic assistance services such as tutoring and math and writing 
centers as well as to personal/social development supports such a counseling and mental health services. 
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course selection decisions; (2) coaching and career advising, to better connect students to 
services and support; and (3) early alerts and predictive analytics, which provide timely 
information to advisors, students, and others when students are at risk of falling off track 
to graduation. Using iPASS tools to perform more than one of these functions 
concurrently better enables colleges to provide sustained, strategic, intrusive and 
in tegrated, and personalized (SSIP) student support. What is more, we posit that when 
multiple functions are integrated through iPASS, they have a greater impact than when 
iPASS tools operate in silos. An integrated approach allows college personnel to better 
collaborate and support students with a more comprehensive understanding of a student’s 
circumstances and performance, both in and outside the classroom. 

We emphasize that the use of technology tools alone does not constitute an iPASS 
intervention. Rather, it is crucial that colleges leverage technology tools to provide 
improved personalized student support. Using an early alert system to email a student 
who did not set up an individualized program plan is not an example of iPASS. But using 
the same early alert system to connect a student with an advisor who then works with the 
student to create an individualized program plan based on the student’s interest is an 
example of iPASS. In this second scenario, not only is an iPASS tool used to identify a 
student at risk, but additional personalized support is provided to the student. 

This paper examines the current state of the literature on iPASS. In particular it 
calls attention to the few studies that inform the field about the potential effectiveness of 
iPASS. The paper is based on an assessment of 60 papers and reports that describe or 
document recent iP ASS-oriented refonn efforts or that report on research conducted to 
better understand the effects of these efforts on student outcomes. It is important to note 
that these papers and reports generally report on technology-mediated interventions that 
perform a single function. In large measure, that is because the iPASS approach is still 


2 Stacey and Karp (2013) elaborated the notion of pervasive supports through the SSIP approach, which is 
rooted both in (1) research showing that challenges to college completion may crop up throughout students’ 
college careers and in non-academic as well as academic domains (Chaplot, Cooper, Johnstone, & 
Karandjeff, 2015) and (2) in research demonstrating that students are often unaware that they need help, 
unwilling to seek it out, or unable to find sources of support (Cox, 2009; Karp, O’Gara, & Hughes, 2008). 
Thus, interventions need to be sustained, to catch students when they need help, and strategic, to connect 
students with the type of support they need when they need it. They also need to be multi-faceted and 
intrusive to be certain that students encounter them. Making non-academic support an integral part of every 
student’s experience means that all students will receive help, even if they think they do not need it. 
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relatively new. Thus many of the interventions discussed in the documents do not involve 
holistic SSIP support that we consider a hallmark of iPASS. 

Still, existing literature on non-integrated or non-holistic technology-mediated 
advising can provide critical insight into the potential impact of the broader approach that 
is iPASS. Therefore, in this paper we include documents that examine iPASS tools that 
support at least one of the aforementioned functions; we also include documents that 
discuss broad iPASS reform efforts more generally. Using this method, we describe what 
kinds of papers and reports have been released about iPASS and closely related 
interventions over the past seven years. We appraise what is known about the potential 
efficacy of iPASS on student outcomes based on the subset of the literature with strong 
research evidence. 

The paper proceeds as follows. We first describe the methods we used to search 
the literature and determine which items are relevant. Next, we provide a general 
overview of the literature available on iPASS, organized by the scope and rigor of the 
studies and other documents examined. Then we review the more rigorous studies as 
determined by criteria rooted in CCRC’s Assessment of Evidence Series (Bailey, Jaggars, 
& Jenkins, 2011) to draw conclusions about iPASS’ potential effect on student outcomes. 
We conclude with suggestions for next steps for assessing the impact of iPASS and 
conducting future research. 


2. Methods 

Our CCRC team looked for relevant iPASS literature in four online academic and 
research-based literature databases: CLIO (online library catalog of Columbia 
University), EBSCO, Google Scholar, and JSTOR. We also searched both institutional 
research websites of early adopter iPASS colleges and vendor websites (see Appendix 
Figures A.l and A.2 for a list of college and vendor websites). We solicited additional 
potential materials through an iPASS grantee listserv. Finally, we asked individual 
vendors for potential items. We limited our literature search to documents published 
since 2010 because iPASS technology and iPASS as a reform has grown and evolved 
exponentially in more recent years. 
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Within the databases, we used search terms (see Appendix Figure A.3) that name 
or characterize iPASS reforms generally (such as “Integrated Planning and Advising 
Services (IPAS),” “Integrated Planning and Advising for Student Success (iPASS),” 
“advising reform + technology”) or that are associated with common elements of iPASS 
(such as “e-advising” and “early alerts”). We also searched these terms in combination 
with terms such as “student outcomes” (e.g., “early alerts + student outcomes,” “iPASS + 
students outcomes”) as we were most interested in understanding the impact of iPASS on 
student outcomes. 

The items that we found in our initial search efforts included journal articles, 
reports, policy briefs, internal documents (accessible on institutional research websites), 
and industry-produced reports. We collected and reviewed roughly 1 10 items that 
appeared by their titles and short descriptions to examine iPASS-oriented reforms. After 
collecting and reviewing all the documents, we created a spreadsheet with individual 
rows for each document and indicated whether or not each document discussed one or 
more iPASS functionalities by labeling items with a “0” or “1” and also indicated 
whether or not they fell into our established timeframe (2010-present). Items that did not 
meet both criteria, such a news article or blog post, were excluded from further review. 

At least two members of our research team then reviewed each of the remaining 60 
documents that matched our criteria, recording notes in the spreadsheet when applicable 
on what intervention was examined, the research questions posed, methods, samples, key 
findings, and strengths and limitations of the research. In addition, using the same criteria 
found in the CCRC Assessment of Evidence Series (Bailey, Jaggars, & Jenkins, 2011), 
we then scored each study that examined student outcomes on a three-point scale 
assessing the rigor of the research method used (See Appendix Figure A.4). 

In the end, we categorized the 60 documents into four different groups based on 
their purpose and rigor: (1) descriptive documents (39 of the 60 items) that describe 
processes and challenges of iPASS implementation, (2) output reports (12 of the 60 
items) that examine usage data to better understand iPASS implementation, (3) 
correlational studies (5 of the 60 items) that examine non-causal associations between 
the different functionalities of the iPASS tools and student outcomes, and (4) rigorous 
outcomes studies (4 of the 60 items) that employ experimental or quasi-experimental 
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methods in a rigorous way to provide causal or near-causal estimations of iP ASS-oriented 
interventions. 


3. Overview of iPASS Literature 

Most of the documents we examined described the ways in which institutions 
implement iPASS technologies and interventions. Very few studies we encountered 
rigorously evaluated differences in student outcomes as a result of an iPASS intervention. 
This likely reflects the newness of iPASS. The literature indicates that most iPASS 
reform efforts are nascent, and institutions have not yet had the opportunity to implement 
comprehensive iPASS interventions, track student-level data from enrollment through 
completion, or assess impact. We would expect to see a growth in outcomes-focused 
studies in the next few years as the field matures and institutions begin to implement and 
evaluate iPASS more systematically. 

Although only studies that focus on outcomes enable one to assess the potential 
impact of iPASS, other types of accounts and studies are still important because they 
describe the context for iPASS reform efforts and help answer questions about the 
specifics of implementing an iPASS intervention. Thus, items in the literature that focus 
on implementation issues may be helpful for understanding the processes involved in 
iPASS technology implementation, such as how technology adoption affects the 
experiences of end users. These accounts often highlight the complexity of implementing 
iPASS technologies in ways that have the potential to affect student outcomes. 

3,1 Descriptive reports 

By and large, the descriptive reports we examined described processes for 
implementing iPASS technologies as well as challenges and potential benefits to end 
users of particular iPASS products. These reports typically included discussions about 
how iPASS technologies were selected and vetted by stakeholders, and they often 
provided insights into how the implementation process could have been improved. Many 
of the descriptive reports described the experiences and lessons learned by the institutions 
in implementing iPASS technologies. 
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The 39 descriptive reports varied in their quality, length, and focus. Several of the 
reports were brief 1-2 page summary descriptions of the institutional process for 
adopting iPASS technologies, sometimes including suggestions for future research 
projects or next steps for the institution. Other descriptive reports provided more in-depth 
discussion about institutional processes and presented findings from end-user and student 
surveys. For example, one report described the perceptions of six administrators at 
Purdue University regarding the implementation of Signals, an early alert technology, 
and the potential benefits that implementing an iPASS intervention could have on student 
success (Arnold, Tanes, & Selzer King, 2010). Similarly, a report from the Virginia 
Community College system described how results from a survey of interviews, coupled 
with some follow-up interviews, was used to help in the implementation of a system-wide 
college planning tool (Herndon, 2011). The report also described how the experience of 
implementing the technology shed light on other processes that could be improved in the 
future. 

A common theme across the descriptive documents is that the implementation of 
an iPASS technology often clarifies technology’s role in a reform effort and often helps 
identify institutional processes that may need to change, such as how long-term academic 
planning is carried out. Although the descriptive documents typically described internal 
processes that are idiosyncratic to individual institutions, personnel from other colleges 
and systems may still find them useful. Indeed some descriptive documents highlighted 
factors that the authors felt other colleges should consider before implementing iPASS in 
order to maximize its potential impact. One report (Pistilli, Willis, & Campbell, 2014) 
indicated that institutions should use data to inform institutional practices when planning 
on how best to implement iPASS technologies. Other reports suggested that colleges 
should seek to better ensure that iPASS reforms meet stakeholder needs by conducting a 
needs assessment of end users prior to implementation (Bradford, 2010; Herndon, 2011; 
Norris & Baer, 2013). Findings from the descriptive documents may inform the next 
wave of iPASS reforms as more institutions begin to think through the best ways to 
implement and encourage the use of iPASS technologies. 
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3.2 Output reports 

The 12 output reports we reviewed focused fairly narrowly on the reporting of 
output data related to iPASS technologies and did not typically describe the 
implementation approach or intervention design in any detail. These documents reported 
on usage of technology, such as the number of times a student logged into particular 
software or the number of early alert flags raised. Output reports also included data on 
student use of support services such as tutoring services or the number of times a student 
met with an advisor. For example, one case study of comprehensive student services 
provided for online students at Lone Star College reported on the number of chat sessions 
attended, number of tutoring sessions attended, and number of early alert flags raised 
(Britto & Rush, 2013). Another report examined usage and opinion survey data collected 
from students using the Degree Compass tool for program planning (Whitten, Sanders, & 
Stewart, 2013). Neither report related these outputs to student outcomes. 

Output reports are helpful for understanding how staff, advisors, and students use 
iPASS technology and the extent to which iPASS tools are being adopted at particular 
colleges. They may thus be useful in identifying potential implementation and user 
adoption challenges and in interpreting or forecasting student outcomes. For example, if 
an output report shows that faculty are not using an early alert system to identify at-risk 
students, one would not expect positive outcomes resulting from the implementation of 
such a system. Note, however, that the output reports we saw did not provide any direct 
evidence about whether the implementation of iPASS technologies led to improvement in 
student outcomes. Nevertheless, output reports can help shed a light on implementation 
fidelity, or the degree to which an intervention is administered as intended. 

3.3 Correlational outcomes studies 

The five correlational outcome studies we examined attempted to infer the 
influence of iPASS on student outcomes such as attendance, grades, and retention. 
However, given the methods used in these studies, the associations between an iPASS 
intervention and the measured outcomes cannot be interpreted as causal. Most of these 
studies did not make a convincing argument about how iPASS interventions alone could 
have caused the identified impact on student outcomes. Methodological challenges 


7 



included small sample sizes, lack of controls for other possible influences on observed 
effects, and the comparison of outcomes among potentially dissimilar groups. 

For example, Arnold and Pistilli (2012) examined the retention rates of three 
cohorts of students using the early alert system Course Signals and compared the 
retention rate of students who enrolled in a course with Course Signals with the retention 
rate of students who did not enroll in a course using Course Signals. The study did not 
control for students’ preexisting characteristics such as demographic background and 
prior academic performance, and it also did not take into account when the students 
enrolled in courses using Course Signals — only that they did enroll at some point during 
their academic career. Control and treatment groups were created post-hoc based on 
whether or not students enrolled in a course with Course Signals — rigorous methods such 
as randomization were not used in the creation of comparison groups. 

Thus, even though this study found a positive correlation between enrollment in 
courses with Course Signals and retention rates, one cannot be sure that use of the iPASS 
tool actually caused the difference. Students in Course Signals and non-Course Signals 
courses may have had different characteristics. Or, professors who chose to use Course 
Signals may represent a different, potentially more student-success focused group of 
faculty compared with professors who chose not to adopt Course Signals in their 
classrooms. 

Lastly, correlational studies focused on a single iPASS functionality, such as early 
alerts, and therefore were not able to estimate the full potential impact of an overarching 
iPASS reform effort. 

3.4 Rigorous outcomes studies 

We define rigorous outcomes studies as those that, while not necessarily causal, 
provide clear evidence that the research is reliable and valid, with estimated impacts that 
can be convincingly li nk ed to the intervention rather than other causes. We include well- 
executed studies with both experimental designs (randomized control trials) and quasi- 
experimental designs — such as regression discontinuity (RD) and propensity-score 
matching (PSM). These kinds of studies are effective in limiting the possibility that 
confounding factors might differ between the two comparison groups. 
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We found only four such studies. We include two of these with reservations 
because they did not provide complete methodological infonnation. For example, one 
policy brief reported using PSM but provided too few details to assess how well the 
design was applied. While we were unable to adequately assess the rigor of two of the 
four studies, we decided to include them, in large measure because there are so few 
studies in this category. We discuss the findings of these studies below in the section 
called Evidence of Impact. 

3.5 Limitations of the literature 

Overall, we found three main limitations to the iPASS literature we examined. 
First, the studies we examined tended to have a narrow focus. For example, while some 
studies attempted to examine the effects of an iPASS technology or intervention 
component on student outcomes, they did so with little consideration of the broader 
iPASS approach described earlier. The drawback from such a narrow focus is that it leads 
to an incomplete test of the efficacy of an iPASS intervention. While iPASS is meant to 
be a holistic approach, encompassing both technology and advising structures and 
processes, most studies examined only one component of a broad intervention effort (by 
focusing, e.g., on whether or not a technology tool was used rather than how it facilitates 
engagement in new forms of advising). 

Second, the literature on iPASS is still rather new and developing, and there is 
little available research on implementation fidelity. To what extent is iPASS implemented 
as intended? Is a particular intervention or array of interventions producing the immediate 
changes in student experience and advisor experience that were planned and anticipated 
under the design? The current literature is mostly silent on these questions. Examining 
implementation fidelity can be an effective means to identify implementation challenges 
and to explore how the intervention may need to be refined. Assessing implementation 
fidelity is necessary to fully evaluate the impact of the reform. If iPASS is not 
implemented well, one should not expect to see improved rates of retention and 
completion. 

Finally, few studies identify if and how iPASS impacts student outcomes 
(Shulock & Koester, 2014). As a whole, the current body of iPASS literature is not very 
useful in connecting the use of iPASS technologies to improved student outcomes, nor 
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does it explain or identify the mechanisms by which iPASS might influence student 
outcomes. A helpful tool to understand the relationship between individual processes and 
outcomes is a logic model, which details each sequential step in an intervention and the 
corresponding actions and effects that might result by it (see Figure 1 below for an 
example). Creating a clear iPASS logic model is a critical exercise for understanding how 
an iPASS reform may affect student outcomes. 


4. Evidence of Impact 

As noted above, the state of the current iPASS literature reflects the nascent stage 
of iPASS reform. Documents focus primarily on understanding how institutions 
implement iPASS tools and initiatives. Additionally, studies which do attempt to attribute 
improved student outcomes to iPASS often fall short methodologically. In this section we 
discuss the four rigorous outcomes studies we found that provide some evidence of 
potential iPASS impact. 

Though the studies in this section are not strictly causal — the observed effects 
they found may have alternative explanations — when taken as a group, these studies used 
a variety of sound methods and data to arrive at similar conclusions. In addition, these 
four studies focused on different functionalities of iPASS tools and found positive 
impacts, which gives us some confidence in our provisional conclusion — though it should 
be noted that none constitute a test of the full iPASS approach given their focus on a 
single functionality. 

Overall, we find preliminary evidence that iPASS-oriented interventions 
positively impact student outcomes, particularly when the interventions emphasize 
personalized student support (Bettinger & Baker, 2014; Civitas Learning, 2014; Lackner 
& Wynne, 2014; Milliron, Malcolm, & Kil, 2014). We would expect similar or even 
larger impacts from the more robust, integrated iPASS interventions now being launched 
around the country. In other words, because most studies examine a single functionality 
of an iPASS tool, such as early alerts or course planning, the effect sizes reported here 
may ultimately provide lower-bound estimates for the potential of a comprehensive 
iPASS refonn effort to improve student success. Furthermore, we posit that there may be 
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a multiplicative effect of iPASS when the intervention addresses more than one function 
to holistically support students at each step in their academic experience (see Karp et ah, 
2016). For example, early alerts and course planning are likely to have a greater impact 
together than if those functions were administered independently. 

The most compelling causal evidence is found in a study of technology-mediated 
coaching. 3 The coaching program, InsideTrack, used a variety of communication 
methods including phone calls and targeted, personalized text messages to help students 
set goals, identify connections between short- and long-term goals, learn self-advocacy, 
and improve time management and study skills. Using a randomized control trial, 
Bettinger and Baker (2014) evaluated the impact of the coaching service on 13,555 
students across eight different postsecondary institutions, including two- and four-year 
colleges in both the public and private, not-for-profit sectors. The participating colleges 
and InsideTrack randomly assigned students across 17 cohorts to a treatment (N= 8,049) 
or control group (N = 5,506). Students assigned to the treatment group received coaching 
services delivered online and by phone for up to 18 months. Coached students persisted 
at significantly higher rates 6, 12, 18, and 24 months following random assignment. At 
both 6 and 12 months, coached students persisted to the next semester at rates 
approximately five percentage points higher than uncoached students. At 18 and 24 
months, coached students persisted at three and four percentages points higher than 
students who were not assigned to receive coaching. 4 

Quasi-experimental studies have also found positive impacts resulting from 
iPASS-oriented interventions. Importantly, like Bettinger and Baker (2014), they found 
that the use of technologies coupled with other personalized student interventions — 
rather than technology on its own — influenced student outcomes. For example, in a study 
of early alerts, Lackner and Wynne (2014) found that students receiving an early alert via 


3 

The study meets the highly rigorous Institute of Education Science’s What Works Clearinghouse (WWC) 
evidence standards (part of the research meets the WWC standards without reservations, and part of it 
meets the standards with reservations). 

4 The five percentage point increase in persistence found in Bettinger and Baker (2014) is similar to effect 
sizes found in a 2008 study of learning communities, a non-iPASS student success initiative (Scrivener et 
al., 2008). Scrivener et al. (2008) found that participants were 5.6 percentage points more likely to be 
enrolled in college three semesters post-program participation. However, this intervention’s impacts were 
not sustained over longer periods of time. A more recent study of learning communities (Weiss, Visher, 
Weissman, & Wathington, 2015) found a positive impact on credits earned, but no impact on student 
persistence. 
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email who also sought out tutoring persisted at higher rates than students who received an 
alert and did not seek tutoring. Lackner and Wynne used a PSM research design to 
control for potential differences in the characteristics of alerted students who did and did 
not seek tutoring. The authors estimated that flagged students who went to tutoring 
improved their likelihood of earning 10 or more credits by seven percentage points (23 
percent versus 16 percent) compared with similar matched students who were flagged but 
did not attend tutoring. 5 Thus, student alerts in combination with individual student 
support improved student outcomes. 

Similarly, Milliron, Malcolm, and Kil (2014) found that early alerts combined 
with personalized contact positively influenced student outcomes. In a series of three 
studies at three institutions, the authors used a predictive analytic tool to identify at-risk 
students and intervene through targeted emails and phone calls. Over the course of 
multiple semesters, the intervention was first refined (for example, by adjusting timing, 
content, or messaging medium) and tested in order to establish an effective approach and 
then tested to estimate its impact. Sample sizes in the colleges were 15,000, 68,000, and 
10,000 students. Each of the three colleges randomized students into control and 
treatment groups. The treatment group received targeted communications from faculty 
while the control group did not receive any intervention. For two of the colleges, a PSM 
approach replaced randomization in subsequent semesters due to operational challenges. 
Comparisons of treatment and control students within colleges found that treatment 
students were between 3.0 and 7.6 percentage points more likely to persist to the next 
term. 6 

In addition to studies highlighting the positive impact of coaching and alert- 
focused interventions on student outcomes, we also reviewed a study that found positive 
apparent effects for an iPASS reform focused on course planning. Austin Community 


5 For context, a study completed at Chaffey College measured the impact of Enhanced Opening Doors, a 
non-iPASS program focused on supporting students on academic probation (Scrivener, Sommo, & Collado, 
2009). After two semesters, program participants earned 2.7 more credits than control group members on 
average. 

6 For comparison, Castleman and Page (2016) examined the impact of the use of financial aid text 
reminders, which is not considered an iPASS tool, because the intervention is not intended as part of a 
holistic approach to student support. Students at community colleges who received text messages were 14 
percentage points more likely to persist to their sophomore year than community college students who did 
not receive the text alerts. The authors did not find a measurable impact among students in four-year 
institutions. 
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College and technology vendor Civitas Learning (2014) used PSM methods to analyze 
students engaged in technology-mediated course planning. The college matched 35,000 
students on characteristics associated with a student’s likelihood to persist, including 
GPA, terms completed, and credits earned. Students who used the degree planning tool 
were 2.4 percentage points more likely to persist compared with matched students who 
did not use the tool. The study also found a positive correlation between how often 
students used the tool and their persistence rates. Students who used the tool 5 or more 
times showed a 7.3 percentage point increase in persistence over similar students who did 
not use the tool 5 or more times during the length of the study (three semesters). 
However, this study does not provide full documentation of research methods; thus, the 
study’s rigor cannot be confirmed. 


5. Discussion and Recommendations 

While much of the literature on iPASS is only descriptive, it does offer limited 
support for the potential effectiveness iPASS. Most of the research focuses on one iPASS 
functionality rather than iPASS as a comprehensive effort. A number of studies provide 
suggestive evidence linking iPASS to improved student outcomes; however, few studies 
have been able to make causal claims about the effectiveness of iPASS interventions. 
Using evidence to determine if and how iPASS supports student success is an important 
next step, with implications for both student outcomes and institutional resources. 

To move from the current largely descriptive state of iPASS literature toward 
more rigorous evaluations of iPASS, we make the following recommendations to 
colleges and researchers who plan to engage iPASS reform and research. 

1. Create an iPASS logic model that illustrates the relationship between iPASS 
tools, interventions, and anticipated subsequent outcomes, and adequately measure the 
inputs, outputs, and ultimate outcomes. In order to better understand the efficacy of 
iPASS, the field needs to explicitly postulate a chain of events that shows how an iPASS 
intervention can lead to improved student outcomes. This could be accomplished through 
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the use of a logic model. A sample logic model detailing how an early alert may improve 
student retention is shown in Figure 1 . 


Figure 1 

Sample iPASS Logic Model - Early Alert System 
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A logic model details the sequential steps that occur in an intervention and the 
corresponding changes in outputs and outcomes that arise from it. A logic model clearly 
articulates the resources that are needed in an intervention — whether human or 
technical — along with a set of activities or actions that are required to administer the 
intervention and understand its impact. Resources for an iPASS intervention (see Figure 
1) may include faculty, technology, and student data, while the activities include those 
actions which initiate the intervention, such as a faculty member identifying and sending 
an early alert flag to a student. The intended outcomes of the intervention include both (1) 
the outputs that describe the direct results of the intended activity (e.g., number of flags 
sent and the number of students who receive and potentially act on those flags), and (2) 
the benefits that are expected to occur as a result of the intervention — positive changes in 
short-tenn and long-term outcomes. In our logic model example, a long-term benefit of 
an early alert intervention is an increase of student retention. 

A logic model strengthens research and practice in two critical ways. First, it 
helps practitioners understand both the resources and processes that are needed to 
adequately engage in a reform effort. By clearly identifying each step in the intervention, 
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colleges can better understand what is required at each step and make refinements prior to 
implementing the iPASS intervention. Often during the implementation process, a college 
will need to reconsider its advising processes. For example, does the college track if a 
student has met with an advisor? Without clearly understanding how an intervention is 
delivered and how students are affected at each step, a college will be unable to identify 
what parts of the intervention are working. 

Second, the logic model can also be used as an evaluation tool to assess whether 
the intervention has met its intended purpose, or to help identify areas for improvement. 
For example, are faculty, staff, and students engaging in the activities prescribed in the 
logic model? Students receiving an early alert about academic perfonnance may, for 
instance, reach out to an advisor but never take subsequent action such as attending 
tutoring to improve their performance. The use of a logic model can help the reform team 
identify and react to this problem. In the case of the tutoring example, the college could 
survey students who did not seek out tutoring after receiving an alert to better understand 
why they did not, and it could then use this information to improve student use of 
advising and tutoring services. A logic model is an important tool colleges can use to 
understand more precisely how an intervention does and does not affect behavior and 
thereby identify weak links in the reform. 

2. Treat iPASS as a holistic approach to advising reform. As colleges implement 
more comprehensive student outreach and support, attempting to examine the impact of a 
single iPASS functionality in isolation is increasingly less practical. Most iPASS studies 
we examined focused on one technology tool coupled with some form of individualized 
student support (e.g., early alert as the technology tool and intrusive advising as the 
improved support). However, iPASS is intended as a broad reform approach integrating 
multiple interventions, which together may have a multiplicative impact on student 
outcomes. Thus, research is needed that attempts to assess the broader impact of a suite 
of technology-mediated supports rather than a single functionality in isolation. 

3. Clearly define and use rigorous research methods. Researchers should clearly 
describe their methods and research design. Most studies we examined did not adequately 


15 



explain how the research was conducted. It is important that researchers make such 
information available. Doing so can help others interpret the strength of the findings. It 
can also help college personnel and others who may be interested in undertaking similar 
research. 

Researchers should also strive to use rigorous methods. Studies employing mixed 
methods, experimental designs (randomized control trial), or quasi-experimental designs 
(e.g., regression discontinuity, difference-in-difference, and propensity score matching) 
can provide the necessary rigor to test for program impact. The What Works 
Clearinghouse operated by the Institute of Education Sciences is a recommended 
resource on designing and delivering studies that meet credible, high standards. 


6. Conclusion 

iPASS is a relatively recent approach, and few studies are available that 
rigorously evaluate its effects. Nonetheless, early findings suggest that iPASS 
interventions may have a positive impact on student outcomes, including persistence and 
credit accrual. However, current research does not examine iPASS as a holistic approach 
to student success but rather focuses on individual iPASS-oriented interventions. 
Additional studies using robust methods are needed to investigate whether iPASS 
improves student outcomes, especially for at-risk populations — including students from 
under-represented minorities and students of low socioeconomic status, who may have 
the most to gain from the reform. Future research should approach iPASS holistically, 
employ rigorous research methods, and clearly articulate how particular iPASS 
interventions are intended to improve student outcomes. 
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Appendix 


Appendix Figure A.l 

Colleges/Universities Institutional Research Websites Searched 


• 

Arizona State University* 

• Queensborough Community College* 

• 

Austin Community College* 

• Ramapo College of NJ 

• 

Austin Peay College* 

• Rio Salado College* 

• 

California Community Colleges* 

• Sinclair Community College* 

• 

California State University, Fresno 

• South Texas College 

• 

Colorado State University* 

• Trident Technical College 

• 

Columbus State Community College* 

• University of Central Florida* 

• 

Community College of Philadelphia 

• University of Delaware 

• 

Dona Ana Community College 

• University of Hawaii* 

• 

Eastern Mississippi Community College 

• University of Maryland* 

• 

Georgia State University* 

• University of Michigan 

• 

Guttman Community College 

• University of Nebraska Lincoln* 

• 

Indiana State University* 

• University of North Carolina at Charlotte 

• 

Ivy Tech Community College* 

• University of South Florida* 

• 

Miami Dade College 

• University of Texas at Arlington* 

• 

Middle Tennessee State University 

• University of Texas at El Paso* 

• 

Montgomery County Community 

College* 

• University of Toledo* 

• 

Morgan State University 

• UT Texas at San Antonio 

• 

Northeast Wisconsin Technical College* 

• Valencia College* 

• 

Northern Arizona University 

• Virginia Community Colleges 

• 

Northern Essex Community College* 

• West Virginia University 

• 

Patrick Henry Community College 

• Whatcom Community College 

• 

Purdue University(Calumet)* 

• Zane State College 


*Achieving the Dream leader schools 
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Appendix Figure A.2 

iPASS Technology Vendor Websites Searched 


• Austin Peay's Degree Campus 

• eAdvisor 

• Blackboard 

• EBI Mapworks 

• Campuscruiser 

• Ellucian 

• CampusLabs 

• Jenzabar 

• Civitas Learning 

• MyEdu 

• Connectedu 

• Sinclair's MAP/SSP 

• Course Signals 

• Starfish/Hobsons 

• EAB 

• Valencia's LifeMap 


Appendix Figure A.3 
Sample of iPASS Terms Searched 


• Advising reform + technology 

• Integrated Planning and Advising for 
Student Success (iPASS) + outcomes 

• College name + predictive analytics 

• IPAS + advising 

• College name + student success + 
technology 

• iPASS 

• College name + early alert 

• iPASS + Gates Foundation + outcomes 

• College name + IPAS (including 
technology assisted advising, full 
spelling) 

• IPAS 

• e-advising 

• Predictive analytics 

• e-advising + iPASS 

• Signals + Purdue 

• Early alert 

• Student success + technology 

• Electronic advising 

• Technology + advising 

• Integrated Planning and Advising System 
(IPAS) 

• Technology + advising + community 
college 

• Integrated Planning and Advising for 
Student Success (iPASS) 

• Technology + advising in higher education 
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Appendix Figure A.4 

CCRC Assessment of Evidence Series: Definitions of Rigor 


Rigor 1 

The pattern of findings could very likely be caused by participant selection effects, 
or by some other factor the author did not take into account. 

Rigor 2 

Findings are fairly sound, even if not necessarily definitive. Alternative explanations 
are possible, but evidence/logic/common sense suggests that they are either 
unlikely, or likely to play a relatively small role. 

Rigor 3 

Findings are highly convincing; it is difficult to think of alternative explanations 
(beyond those the author convincingly rules out.) 
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